Debugging in the Extreme: Spectrum-based Fault Localization with Limited Test Cases

Spectrum-based Fault Localization (SBFL) is a well-known debugging technique that locates fault in program code by utilizing execution profiles (spectra) of pass and fail test cases. Hence, the performance of SBFL depends on the test cases executed and the test results. In the most extreme scenarios, the debugging process may have to be conducted with only one fail test case, one pass test case or no pass test case. These scenarios might occur due to extremely high or extremely low failure rates or when software testers decide to stop running more test cases due to time and resource constraints. However, limited test case execution profiles may reduce the accuracy of SBFL metrics. In view of this, we evaluate the performance of SBFL metrics in these extreme scenarios to identify the best performing SBFL metric for each of these scenarios. From the experiment results, we have further discovered the convergence in performance for SBFL metrics under these extreme scenarios.